Telecommunications network traffic metrics evaluation and prediction

ABSTRACT

A method for evaluating and predicting telecommunications network traffic includes receiving site data for multiple geographic areas via a processor. The processor also receives weather data, event data, and population demographic data for the geographic areas. The processor also generates predicted occupancy data for each of the geographic areas and for multiple time intervals. The processor also determines a predicted telecommunications network metric for each of the geographic areas and for each of the time intervals, based on the predicted occupancy data.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.16/433,434, filed Jun. 6, 2019, and titled “Telecommunications NetworkTraffic Metrics Evaluation and Prediction,” which claims priority to andthe benefit of U.S. Provisional Patent Application No. 62/681,371, filedJun. 6, 2018 and titled “Telecommunications Network Traffic MetricsEvaluation and Prediction,” the disclosure of each of which is herebyincorporated by reference in its entirety.

FIELD

The present disclosure relates to the field of telecommunicationsnetwork planning, management, and operation.

BACKGROUND

There are 336 cities in the world with populations that exceed 1million, and 75% of the world's population will live in urban areas bythe year 2050. In such cities, there can be large daily migrations ofthe cities' inhabitants. For example, in the U.S. alone, there are 20cities in which more than 50% of the inhabitants commute or otherwisemove within the city on a typical day. With the proliferation ofconsumer software applications having varying bandwidth and latencyspecifications, the demand on associated communications networks canvary significantly with fluctuations in the number of active users andthe type of applications being used.

SUMMARY

Embodiments of the present disclosure facilitate the forecasting ofvariations in telecommunications traffic (e.g., relative to areference/base level of expected telecommunications activity for aspecified time, date and/or location, or average usage for a specifiedtime), for example in densely populated areas and/or in areas havingsignificant daily migrations of people. In some such implementations,external data and machine learning techniques are used to predict spikesin, and/or sudden changes on, telecommunications network traffic demand.Alternatively or in addition to forecasting/predictingtelecommunications traffic variation and/or telecommunications trafficdemand, systems and methods of the present disclosure can facilitate theforecasting of telecommunications network metrics such as (but notlimited to): a type of traffic, an amount of traffic, a volume oftraffic, a network latency, a packet drop, traffic variation compared toa reference point (relative), Such forecasts/predictions can be providedto one or more operators (e.g., via a graphical user interface (GUI)),who may use such insights to 1) improve their network resourceallocation to increase network performance, 2) improve energy efficiencyby decreasing unused network resources, and/or 3) perform short-term andlong-term network capacity planning. Alternatively or in addition,systems and methods of the present disclosure can be used to generaterecommendations, for presentation to operators (e.g., via the GUI),based on the forecasts/predictions generated by the system.Recommendations can include one or more of: network architecturemodification recommendations, network traffic routing modifications,network capacity planning recommendations, network equipmentmodification/replacement recommendations, etc.

In some embodiments, a method for evaluating and predictingtelecommunications network traffic includes receiving site data formultiple geographic areas via a processor. The processor also receivesone or more of: event data, historical data, predictive data (e.g.,relating to weather and/or traffic), live data (e.g., relating to socialmedia data and/or transportation services), and static data (e.g.,relating to population demographic data, number of businesses and/orvenue capacity) for the geographic areas. The processor also generatespredicted occupancy data for each of the geographic areas and formultiple time intervals. The processor also determines a predictedtelecommunications network metric for each of the geographic areas andfor each of the time intervals, based on the predicted occupancy data.The method can also include generating, via the processor and for eachgeographic area of the plurality of geographic areas and for each timeinterval of a plurality of time intervals, predicted activity categorydata, and the determining the predicted telecommunications networkmetric can further be based on the predicted activity category data.Alternatively or in addition, the method can also include receiving: (1)an indication of a first geographic area of the plurality of geographicareas and (2) an indication of a first time interval of the plurality oftime intervals, via a graphical user interface (GUI), and sending asignal to cause display, within the GUI, of the predictedtelecommunications network metric associated with the first geographicarea and the first time interval, in response to receiving theindication of the first geographic area of the plurality of geographicareas and the indication of the first time interval of the plurality oftime intervals. Generating the predicted occupancy data can be based onthe site data, the weather data, the event data, and/or the populationdemographic data. The site data can include site occupancy capacitydata.

In some embodiments, a method for evaluating and predictingtelecommunications network traffic includes receiving site data for apredetermined geographic area and event data for the predeterminedgeographic area via a processor. The processor generates a predictedtelecommunications traffic metric, for the predetermined geographic areaand for a predetermined time interval, based on the site data and theevent data. The processor can send a signal, for example using anapplication programming interface (API) and/or a comma-separated values(CSV) file, to cause display of the predicted telecommunications trafficmetric via the GUI, for example in response to receiving an indicationof the predetermined geographic area and an indication of thepredetermined time interval via a graphical user interface (GUI). Thepredicted telecommunications traffic metric can include a predictedtraffic volume and an associated traffic type. The event data caninclude movement data associated with an individual. The method can alsoinclude sending a signal (e.g., via the API and/including transmissionof a CSV file) to cause display of a map including the predeterminedgeographic area within the GUI.

In some embodiments, a method for predicting resource demand includesreceiving, via a processor, event data for a geographic area. The methodalso includes receiving capacity data for a venue within the geographicarea, and determining a population volume time series based on the eventdata and the capacity data. The method also includes receivinghistorical data associated with the geographic area, and determining arelativized population volume time series based on the population volumetime series and the historical data. A resource demand for thegeographic area is then predicted based on the relativized populationvolume time series.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a plot of performance characteristics for a variety oftelecommunications applications.

FIG. 2 is a system diagram showing components of a system for evaluatingand predicting telecommunications network traffic, according to someembodiments.

FIG. 3 is a process flow diagram illustrating a method for evaluatingand predicting telecommunications network traffic, according to someembodiments.

FIG. 4 is a process flow diagram illustrating a method for predictingresource demand.

FIG. 5A is a process flow diagram for a resource prediction system,according to some embodiments.

FIG. 5B is a process flow diagram showing an application of the processof FIG. 5A, according to an implementation.

DETAILED DESCRIPTION

In urban areas, people move/travel along routes that can correspond toor be associated with any of a wide variety of activities and/orobjectives. Depending on the location, time, and type of activity, anindividual can seek any of a wide range of services or resourcesincluding (but not limited to) wireless network (e.g., Mobile, WiFi®,etc.) connectivity, transportation, energy/power, food/drink, emergencyservices, safety, etc. As demand for such services changes, providers ofthose services may need time to make adjustments to their supply chain,leading to inefficiencies in delivery of the services. Known methods ofurban resource allocation are typically based on demand forecastsgenerated using historical data. Such approaches, however, rely onlimited data that lacks context, and produce low-accuracy, averaged-outpredictions that do not facilitate differentiation between anomalous andspontaneous real-world behavior. As a result, there is generally a gapbetween actual, “real world” demand and forecast demand for a givenbusiness vertical. To address this gap, embodiments set forth hereinprovide an improved-accuracy, flexible demand prediction andoptimization platform that takes into account population, location, andpredicted activity and/or route of traversal (i.e., the path along whichan individual walks or drives). System-level implementations of thepresent disclosure can include a knowledge-base; a memory storingprocessor-executable steps such as extraction, transformation,enhancement, and loading of data; multiple different machine learningalgorithms and/or optimization algorithms; and an intuitive datadelivery platform. Systems set forth herein can be specifically designedto comply with applicable U.S. and EU data privacy laws. In someembodiments, the system can generate recommended actions and/orremediation measures in response to predicting a high demand (e.g.,above a predetermined threshold demand). Recommended actions andremediation measures can include, but are not limited to, one or moreof: reallocation of resources (e.g., network resources, computingresources, etc.), restriction of already-allocated resources (e.g.,network resources, computing resources, etc.), re-prioritization ofalready-allocated resources (e.g., network resources, computingresources, etc.), addition of resources (e.g., network resources,computing resources, etc.), alerting/notifying one or more users, etc.

For telecommunications network operators, satisfying the Quality ofService (QoS) promised to their customers is crucial. Conventionally,operators try to add as much capacity as possible to their networks, tobe able to handle an anticipated worst-case scenario. Such strategies,however, are expensive, energy inefficient, performance inefficient, notscalable in larger cities, and not sustainable for 5G systems. Theadoption of virtualized networks and the ability to dynamicallyadd/remove network and compute resources, has made it possible toimprove this strategy by implementing a more dynamic network resourceallocation. Networks are often designed and managed “application blind,”i.e., network operators only have access to Layers 0-3 of the network,and thus are unable to access software application information.Furthermore, due to customer privacy concerns, network operatorstypically cannot collect data from individuals on their mobile activity.As such, it is difficult for operators to forecast how network users ata large scale will use the network and how much resources they willneed.

Attempts at predicting demand using historical layer 0-3 trend data haveoften produced low accuracy predictions, for example since (1)aggregated traffic information drastically varies based on theapplication used by a majority of customers, (2) trend analysistechniques using aggregated network usage typically cannot predictspikes and sudden variations, and/or (3) there are new applications thatare continuously being introduced into the network(s) with differingbandwidth and latency requirements, for which historical data may notexist. This lack of data on network user behavior can result ininefficient networks and/or a poor user experience. Moreover, the issuesof variation and unpredictability in network demand may be exacerbatedby the introduction of 5G wireless systems, since 5G will accommodatenew sets of applications that can span up to 4 orders of magnitudedifference in bandwidth and latency requirements, as shown in the plotof FIG. 1. More specifically, FIG. 1 is a plot of performancecharacteristics (typical bandwidth throughput and communications delay)for a variety of telecommunications applications, such as monitoringsensor networks, virtual reality, augmented reality, bidirectionalremote controlling, autonomous driving and video streaming. Thisapplication-based variability in network application usage canoverburden the network, leading to performance problems.

Embodiments of the present disclosure can be used to improve theallocation of network systems, by more accurately forecasting/predictingdemand for such resources. Such forecasts/predictions can be generatedbased on population data, location data, and/or predicted activityand/or route of traversal, together with one or more machine learningmodels trained over one or multiple industry verticals. Examples ofindustry verticals for which demand can be predicted include, by way ofnon-limiting example: transportation (such as ride hailing and publictransportation), mobile telecommunications, mobile internet, food andbeverage, emergency services (such as police, firefighter, paramedics,ambulance, and public safety services), and retail (e.g., sale ofmerchandise). Models set forth herein can be autonomous in nature (i.e.,self-training) and/or can be configured to generate “first impression”demand forecasts/predictions for new markets for which no trained modelyet exists. Alternatively or in addition, models set forth herein can beconfigured to predict correlations between real-world phenomena andvarious business demands. Real-world phenomena can include city eventssuch as conferences, trade shows, exhibitions, sporting events,transportation delays/shutdowns, weather events, etc. Business demandsthat can be impacted by the real-world phenomena include, by way ofnon-limiting example, mobile internet services, transportationsservices, public safety resources, emergency services, food and beveragedemand/sales, and/or merchandise demand/sales.

In some embodiments of the present disclosure, a system for moreeffectively and/or efficiently evaluating and predictingtelecommunications network traffic includes a software platformconfigured for use by an operator of a telecommunications network. Thesystem can be implemented in a computing device, and may be disposedwithin a premises of the network operator or made accessible to thenetwork operator via a cloud computing network (“the cloud”). The systemcan be accessible, for example, via a user interface that may be graphicor text-based and/or via API.

Embodiments of the present disclosure can provide benefits totelecommunications network operators as well as other actors (such asend-users of the network). For example, in some implementations,information used by the system to generate network performancepredictions are obtained through available public or statisticalresources without violating a network user's privacy. Systems set forthherein can provide telecommunications operators with unique access todata about their customers/users that can greatly enhance theiroperation and business. For example, by predicting large spikes andvariations in network usage, systems of the present disclosure make itpossible for network operators to proactively tune and optimize theirnetwork performance to meet their QoS targets during such events, and/orto improve energy efficiency by decreasing network resources when theyare not needed (e.g., putting small-cell base stations on standby toreduce energy usage). Knowledge about time/location-specific trafficincrease and decrease trends can allow telecommunications operators toimprove their short-term and long-term capacity planning and improvereturn on capital investments. Using systems and methods set forthherein, network operators can also assess their network performance andperform risk analysis, for example by monitoring (e.g., using theinformation layers described herein) variations in network usage,changes in weather conditions, electricity shortages and/or constructionevents. Based on these information, the invention can predict outagetype, location and time.

Embodiments set forth herein include a prediction and optimizationplatform that combines a knowledge base (e.g., including populationlocation and activity data) with customer and domain-specific data toperform high accuracy prediction across various business verticals ordomains. The platform leverages a collection/set or “portfolio” oftrained machine learning models configured to perform spatial andtemporal predictions over any domain that is impacted by populationlocation and activity. The platform can identify correlations amongmultiple different business resource demands, external phenomena, andfuture trends, and can respond to anomalous behaviors/event that couldnot be predicted/forecasted using a machine learning model for a singlebusiness domain alone.

Knowledge Base

Urban areas can be uniquely identified based on one or more of a widevariety of characteristics, including, for example, neighborhood,architecture, history, seasonal activities, and culture of theinhabitants. However, there can also be significant similarities amongthe cities, countries, or continents in which people gather, and amongthe activities those people engage in. Systems set forth herein caninclude a self-evolving knowledge-base (e.g., an autonomous machinelearning model) that can detect and capture/store unique characteristicsof urban areas, for example characteristics related to population,location, predicted activity and/or predicted activity category.Examples of activity categories that can be predicted include, but arenot limited to: internet activities, transportation activities (e.g.,vehicular traffic), wireless network communications related activities,energy/power related activities, social activities, emergency servicesrelated activities, safety-related activities, etc. The knowledge-baseis an automated, highly accurate, cost-effective, dynamic, secure, andeasily scalable platform that continuously collects data from varioussources in the following categories, optionally for each of multipleurban centers:

-   -   Event data: Data associated with organized events including        sporting events, concerts, parties, conferences, workshops,        cinemas, tradeshows, meetups, etc.    -   Historical data: Historical visitor statistics (e.g.,        attendance) for areas of interest such as: bars, cafés, night        clubs, hospitals, airports, train stations, etc., for example as        associated with a particular day of the year, season, date, etc.    -   Predictive data: Weather data, traffic data, etc.    -   Live data: Social Media data, transportation data, etc.    -   Static data: number of businesses within a specified geographic        region, types of businesses within a specified geographic        region, demographic data, etc.

Extract, Transform, Enhance, and Load Platform

In some embodiments, a processor-implemented system can leverage one ormore big data tools (such as Apache Spark or Postgres Database) toextract one or more large data sets (e.g., data sets having a size of atleast about 1 gigabyte (GB) or between about 1 GB and about 1 terabyte(TB) from its knowledge-base (also referred to herein as a “datawarehouse”), for example via an automated process. Upon extraction ofthe large dataset(s) from the data warehouse, the data from the largedataset(s) can be transformed from a first format into one or more newformats and modified/enhanced such that it includes one or more newfeatures (i.e., a “feature enhancement” process), thereby producing atransformed dataset. The feature enhancement processes can include apredetermined sequenced set of multiple supervised machine learningalgorithms and/or unsupervised machine learning algorithms to achievehigh prediction accuracy, for example across multiple different businessapplications (e.g., large-scale performances, political events, sportevents, city parades, national holidays, social events, tradeshows,exhibitions, conferences, etc.). Three examples of such models are:

Category Classifiers

Information/data associated with historical organized events can begathered from various sources, however, not all such information/data iscategorized at the source. In some embodiments of the presentdisclosure, an event classifier platform (implemented in software and/orhardware) includes or uses one or more Natural Language Processing (NLP)models. NLP models classify each event within an associated category,and the association between the classified events and their designatedcategories can be stored (e.g., in a table) in memory. Eventclassification can serve to improve the prediction accuracy of one ormultiple different business applications, for example by providinginter-operability within the knowledge-base.

Duration Estimators

In some embodiments, machine learning is used to detect correlationsbetween a pattern of increase or decrease to population or populationdensity, and one or more organized events. One or more of multipledifferent factors, such as event type, event category, event location,etc. can impact the amount of change and/or rate of change in populationor population density around a given venue. Machine learning models ofthe present disclosure can use stochastic analysis and optimization toprecisely estimate the timing and duration of one or more peaks/spikesin population change around each of multiple venues/neighborhoods, thusproviding useful data as input to one or more business applications.

Attendance/Capacity Estimator

In some embodiments, machine learning algorithms and big-data tools areused to maintain an accurate measurement of maximum capacity andexpected attendance over time for any of a wide range of venues andevents. These measurements can then be used/implemented in the accurateprediction of population density flux in areas hosting both organizedevents and non-organized events.

Once the feature enhancement process has completed, and the transformeddataset has been generated, it can be loaded onto a low latency,scalable, high-availability database. As used herein,“high-availability” refers to a characteristic of a database or othersystem that a measurable level of operational performance thereof (e.g.,uptime or other service metric) remains consistent over a period of timethat is longer than has historically been sustained. High-availabilitycan mean the condition of being operational at least about 99.999% ofthe time, or at least about 99.995%, or at least about 99.99%, or atleast about 99.9%, or at least about 99.8%, or at least about 99.5%, orat least about 99%, or at least about 98%, or at least about 97%, or atleast about 95%, or at least about 90%, or at least about 55.5555555%.This database can serve as a foundation for providing key insightsacross several business applications using the predictive modelsdiscussed below.

Predictive Models for Diverse Set of Domains

Prediction platform technology systems (PPTs) set forth herein can beused to predict (and, hence, plan for) resource demand that varies(whether in predictable/foreseeable ways or unexpectedly) with time andlocation. The PPTs can provide such predictions across a wide-range ofbusiness verticals (or “domains”) through a portfolio of trained modelsusing system data and/or customer data. Depending upon theimplementation, PPT embodiments set forth herein can perform one or moreof:

-   -   Prediction of resource demand in new domains for which no        trained models exist: Predictive methods can be based on deeply        correlated data, for example related to population location        and/or activities in urban areas, which often directly impact        businesses and industries (e.g., sales traffic, revenue, etc.).        Once initial models have been trained and established,        predictions and associated insights can be generated across a        wide range of industries. In addition, each particular        prediction can be used to inform at least one further prediction        for a related variable/value. For example, trained models for        predicting demand for transportation and network connectivity in        urban regions for specified timeframes, and/or the output of        such trained models, can be used to perform predictions of        demand for food demand/supply, demand for staffing, demand for        emergency services, demand for energy, etc. Each time a model is        used to make predictions for a new industry domain, the accuracy        of the statistical inference capability of that model can        improve (e.g., via feedback from or integration with model data        from another and/or retraining based on ground truths) without        forcing an unsustainable or dramatic increase in that model's        complexity. Over time, a set of multiple domain-specific        predictive models can be compiled such that, when used in the        aggregate, their collective predictive power and accuracy        (and/or the predictive power and accuracy of each        domain-specific model) are synergistically improved. An example        method for predicting resource demand is shown in, and described        with reference to, FIG. 4.    -   Correlation of phenomena across business applications: Using        multiple different domain-specific predictive models (e.g.,        covering multiple different business applications), correlations        can be detected or calculated between a given phenomenon or        event (e.g., a number, type and/or category of event(s) within a        specified period of time) and business applications, thus        facilitating the improved prediction of increases/decreases in        future performance (e.g., supply, demand, revenue, traffic,        etc.) of the business applications. As such, correlations        between, for example a number of events (e.g., sports events)        and a particular resource demand, or between weather and        resource demand, or between traffic and demand can be        determined/detected. The multiple different domain-specific        predictive models can be combined with locally-stored location        data and/or activity data to perform prediction across domains        for behavior that otherwise would seem anomalous and        unpredictable from within each isolated domain. In other words,        sudden changes in population movement that may be classified as        anomalous by a machine learning algorithm trained within a        specific industry domain alone, may instead be recognized as        expected (e.g., seasonal) by a composite/combined statistical        model generated based on models for each of a plurality of        different business/industry verticals (or domains), such as        transportation and networking.

In some embodiments set forth herein, a system for generatingpredictions/forecasts (e.g., of resource demand) includes multiplesupervised machine learning models each configured to generatepredictions/forecasts for an associated business/industry vertical ordomain, Inputs to the machine learning models can include customer data.In some implementations, multiple different machine learning models(some or all of which may be operating in parallel) are used aspredictors (optionally in combination with or based on customerhistorical data and/or external contextual data) for new demandforecasting models (which also optionally take into account the customerhistorical data and/or external contextual data). Algorithms of themachine learning models can include (but are not limited thereto) one ormore regression models and/or one or more neural networks, such asfeed-forward neural networks, convolutional neural networks (CNNs),artificial recurrent neural networks such as long short-term memory(LSTM), etc. In some implementations, each domain-specific supervisedmachine learning model uses a different machine learning algorithm, orthe machine learning algorithm for each domain-specific supervisedmachine learning model is selected based on the associated domain.

Business Optimization Platform

In some embodiments, in addition to generating accuratepredictions/forecasts of resource demand, systems of the presentdisclosure can also generate, as outputs, recommended actions foradapting a supply chain to adapt to or accommodate temporal and/orspatial demand variations. Each business application can have numericaland implementation attributes that differ from the numerical andimplementation attributes of other business applications. Examples ofnumerical and implementation attributes include, but are not limited to:bandwidth size, positioning, and/or adjustments thereto (e.g., in thecontext of telecommunications networks), fleet size, such as number ofvehicles/cars and adjustments thereto (e.g., in the context oftransportation), number of employees, equipment and/or vehicles, andadjustments thereto (e.g., in the contexts of police, paramedic,ambulances, and emergency rooms), and quantity of line items (e.g., forfood in the contexts of restaurants and grocery stores).

A Fundamental Optimization Problem

Although the implementation of optimization actions can differ acrossbusinesses or business applications, the mathematics underlying suchoptimization actions can be categorized as either single-objective ormulti-objective stochastic optimization. For example, businesses thatrely on population density flux can face similar stochastic optimizationproblems, and as such, similar solutions can be drawn from a range ofqueueing and scheduling models related to operations research andresource allocation. Business solutions from the area of operationsresearch can fall into one or more of three categories: stochastic batchscheduling, multi-armed bandit scheduling, and queueing systemsscheduling. Depending on the particular business application andattributes of that business application, a machine learning model can beselected and implemented, for example based on associated measures offitness to the real-world situation and/or accuracy (which may beinformed by historical data). Resource allocation solutions topopulation density flux problems can take the form of one or moremixed-integer programming models (with optimization under linear andbounded constraints, for example) and/or solutions to various forms ofthe Knapsack problem in combinatorics.

The accuracy of massive-scale or large-scale optimization problemsinvolving the location and activities of large populations can sufferfrom incomplete information. Knowledge bases set forth herein, however,provide a solution to information completeness issues, since underlyingstochastic scheduling is performed via a Bayesian inference process thatupdates some or all probability hypotheses as additional information isreceived or becomes available via an data input stream. Illustrativeexamples for two massive-scale markets, mobile networks andride-hailing, are presented below.

Mobile Networks

Mobile networks typically include a front-end (for radio and signalprocessing) and a backend (for network routing and connectivity to datacenters). Front-ends of 5G network equipment can be adjusted, improvedor optimized by leveraging the Cloud and/or via virtualization of RadioAccess Networks (RANs). In next generation of RANs, a pool of resourcesincluding Base Band Units (BBUs) are provided to a range of microcellsthat support 3D Beamforming and Massive multiple-input andmultiple-output (MIMO). Optimizing resource allocation in such pools ofresources can help to ensure high performance and cost-effectiveoperation of 5G.

In the backend, Software Defined Networks (SDNs) are known communicationroutes/paths that can transmit customer data to data centers. Improvingor optimizing distribution of the traffic over the available SDN pathscan significantly reduce transmission times and delays, which could havesignificant benefits for a wide variety of 5G applications such asvirtual reality (VR), augmented reality (AR), self-driving cars,telemedicine, critical internet-of-things (IoT), etc. The underlyingmodels for achieving such improvements/optimizations of trafficdistribution can be part of the operations research and resourceallocation domains described herein. Mobile networks can use MulticlassQueueing Networks (MQNs) for the scheduling of queueing systems in whichthe tasks/jobs to be completed arrive at varying and/or random timeintervals. Systems of the present disclosure can generate predictionsand recommended actions not only the scheduling of such randomlyarriving tasks/jobs, but also for jobs whose completion time as well ascost rate incurred per unit time are randomly distributed, such as canarise in the context of ride-hailing systems (discussed below). In viewof the large capital and operational investment that are typicallyemployed by organizations for network infrastructure and its operation,significant benefits can be achieved by even modest improvements in thereallocation of idle and unused resources throughout the mobile network.

Ride-Hailing Systems

Ride-hailing companies can impose adjustments to their supply chains by,for example: 1. providing driver incentives, which can be temporalincentives and/or spatial incentives, to change the fleet size and/or tomove the fleet, and 2. adjusting passenger pricing such that it isdependent on the time of arrival. Global optimization/accommodation ofride-hailing demand can be achieved by the multi-domain machine learningsystems set forth herein, through the generation of recommended actionrelating to driver incentives and/or passenger pricing adjustments, toincrease car/taxi availability and, consequently, ride-hailing revenue.

Intuitive Data Delivery Platform

In some embodiments, a system includes a platform (implemented inhardware and/or software) that can service multiple different domainsusing user interfaces (e.g., graphical user interfaces (GUIs)) andendpoint application programming interfaces (APIs). The platform canidentify key characteristics of customer prediction requests and/orcustomer data, for example, via a survey (which may be presented to auser, e.g., via the GUI, and/or sent to a user via the API and/or sentin the form of a CSV file). In the backend of the system, data cleaningand feature enhancement can be performed, and the type of external data,trained models, and machine learning (ML) algorithms can bedetected/identified that will provide the highest achievable accuracyprediction in that domain.

FIG. 2 is a system diagram showing components of a system 200 forevaluating and predicting telecommunications network traffic, accordingto some embodiments. The system 200 includes a predictor 208 in networkcommunication (e.g., via wireless network 101) with one or more computedevices 210 (e.g., third-party servers, etc.) and one or more operatorcompute devices 212. The predictor 208 includes a memory 202 operablycoupled to a processor 204, and the processor 204 is operably coupled toa communications interface 206. The memory 202 stores data 202A andpredictions 202B. The data 202A can include, for example, data receivedfrom the one or more compute devices 210 (e.g., via wireless network101), for example one or more of: site data 214A, weather data 214B,event data 214C, and population demographic data 214D. Site data 214Acan include, but is not limited to, one or more of: property layoutdata, facility type data, facility size data, facility usage data,actual site occupancy data, and site maximum occupancy data. Weatherdata 214B can include, but is not limited to, one or more of:temperature, weather description (sunny, cloudy, partly sunny, partlycloudy, mostly sunny, mostly cloudy), wind, barometric pressure,precipitation, and humidity. Event data 214C can include, but is notlimited to, one or more of: individual movement data (e.g., GPS locationdata), event type data, event duration data, event attendance forecastdata, and actual event attendance data. Population demographic data 214Dcan include, but is not limited to, one or more of: age data, incomedata, gender data, predicted device type data, profession data,educational level data and social media usage data. Each of site data214A, weather data 214B, event data 214C, and population demographicdata 214D can include data that is specific to a predetermined/specifiedgeographic location and/or specific to a predetermined/specified day,date, time, and/or time interval.

The processor 204 can access data 202A and generate predictions 202Bassociated with one or more telecommunications network metrics based onthe retrieved data 202A. Examples of telecommunications network metricsthat can be predicted by the processor 204 of the predictor 208 caninclude, but are not limited to, one or more of: total traffic volumefor a predetermined time interval, traffic type, and traffic volume perunit time, total bandwidth variation for a predetermined time interval,and latency distribution for a predetermined time interval.

The predictor 208 can receive, e.g., via its communications interface206, a message from an operator compute device 212 (e.g., via wirelessnetwork 101) including an indication of a geographic area and/or anindication of a time interval of interest (216). In response toreceiving the message 216, the predictor 208 (e.g., via processor 204)can perform a lookup or otherwise retrieve (e.g., from memory 202) oneor more predicted telecommunications metrics associated with theindication of a geographic area and/or the indication of a time intervalof interest (216), and send the one or more predicted telecommunicationsmetrics via a message (218) to the operator compute device 212.

In some implementations, data stored in the memory 202 includes multiple(e.g., 3) layers of data/information, for example organized as follows:

1. A Location Layer can include a dynamic “map” (e.g., defined within agrid) that can automatically adapt itself to a change in networkarchitecture and/or to the demands of a user/operator. For elements ofthe map (e.g., meshes of the grid), data associated with multiple typesof buildings and/or streets (e.g., Building Directory data,International Building Code (IPC) data, etc.) can be received or“gathered” from one or more data providers. For each building and/orlocation, a maximum occupancy capacity, in terms of people and/or numberof devices that can be connected to a network, can be determined.

2. A Human/Activity Layer can include data such as a number of occupantsper unit time and/or per activity (future projected and/or actualhistorical). The data can be gathered using one or more of: internetcrawling, Public APIs (such as Google Places API, Facebook events API,etc.), or human activity apps APIs (e.g., health-related apps). Oncedatasets for multiple different categories of building or other locationtype have been gathered, a machine learning engine is built for eachcategory, and trained, for example, using the following inputs togenerate outputs:

Inputs

1. Weather:

-   -   a. Temperature    -   b. Wind    -   c. Clear/Rain/snow

2. Event category

3. Human archetype

-   -   a. Age

4. Time info:

-   -   a. Hour    -   b. Day of the week    -   c. Month

Outputs

1. Number of occupants

2. Activity category type

3. A Network Layer can include network usage data for each location andtime, generated based on the outputs from Layer 2 (i.e., the associatednumbers of occupants and activity category types). The Network Layer canbe said to have a “knowledge base” that includes statistical informationrelated to network usage for different activities, such as voice,browsing, video streaming, gaming, virtual reality, etc. The knowledgebase can be generated using data provided by one or more mobile networkperformance systems and/or existing statistical models. For example,using one or more datasets of the knowledge base, a machine learningengine can be trained to predict network usage for each location andtime, using the following inputs to generate outputs:

Inputs

1. Number of occupants

2. Location category

3. Activity category

Outputs

1. Network usage

-   -   a. Voice    -   b. Backup    -   c. Game    -   d. Video streaming

2. Network usage

-   -   a. Bandwidth    -   b. Latency

In some embodiments, by combining the foregoing layers ofdata/information, a system of the present disclosure can providereal-time data, using real-time data and/or prediction data and trainedmachine learning engines. Machine learning engine implementations caninclude, but are not limited to: regressions, classifications, neuralnetworks, and Hidden Markov models. In other embodiments, predictionscan be generated and provided by the system exclusively using real APIdata, without the use of machine learning models.

FIG. 3 is a process flow diagram illustrating a method 300 forevaluating and predicting telecommunications network traffic, compatiblewith the system 200 of FIG. 2, according to some embodiments. As shownin FIG. 3, the method 300 begins upon receipt (e.g., at a processor suchas processor 204 of FIG. 2) of one or more of: site data for eachgeographic area of a plurality of geographic areas (at 320), weatherdata for each geographic area of a plurality of geographic areas (at322), event data for each geographic area of a plurality of geographicareas (at 324), or population demographic data for each geographic areaof a plurality of geographic areas (at 326). At 328, predicted occupancydata is generated for (1) each of the geographic areas, (2) for aplurality of time intervals, for example based on the received one ormore of site data, weather data, event data, or population demographicdata. Based on the predicted occupancy data, one or more predictedtelecommunications network metrics are determined for (1) each of thegeographic areas, (2) for a plurality of time intervals. In someimplementations (not shown), the method 300 can further includegenerating recommendations, for presentation (e.g., via a GUI) to one ormore network operators (sent, for example, via an API and/or in the formof a CSV file), based on the one or more predicted telecommunicationsnetwork metrics.

FIG. 4 is a process flow diagram illustrating a method 400 forpredicting resource demand. As shown in FIG. 4, event data for ageographic area or region is received, via a processor (such asprocessor 204 of FIG. 2), at 440. At 442, capacity data for a venuewithin the geographic area is received via the processor. A populationvolume time series is generated/determined at 444, based on the eventdata and the capacity data. At 446, historical data associated with thegeographic area is received via the processor. A relativized populationvolume time series is determined based on the population volume timeseries and the historical data, at 448. Then, at 450, a resource demandis predicted for the geographic area based on the relativized populationvolume time series. Any of the steps of method 400 can be performed via,or include an interaction with, a GUI, an application programminginterface (API) and/or a comma-separated values (CSV) file.

FIG. 5A is a process flow diagram for a resource prediction system,according to some embodiments. As shown in FIG. 5A, the process 500Aincludes receiving data, at 560, from one or all of multiple sources (A)through (E), including one or more of: real data (A), live data (B),historical data (C), predictive data (D), and static data (E). At 562,an intermediate, first prediction stage occurs, in which one or more ofthe inputs (A) through (E) received at 560 are used to predict valuessuch as event attendance (562A), one or more telecommunications metrics(562B), and one or more event categories (562C). The predictionsperformed at 562 can include, for example, any of the methods discussedbelow. Based on the predictions generated during the first predictionstage 562, one or more further (second) predictions 564 can begenerated. The predictions 564 can be predictions of resource demandsfor a specified event (e.g., the event for which attendance waspredicted at 562A and/or that was categorized at 562C), and can beperformed using, for example, any of the methods below. Resource demandpredictions generated at 564 can be optionally fed back to theintermediate stage for use in generating further intermediatepredictions, whether for the present event or for one or more futureevents.

FIG. 5B is a process flow diagram showing an application of the processof FIG. 5A, according to an implementation. As shown in FIG. 5B, theprocess 500B is predicting the impacts on various resource demands(namely telecommunications, traffic, and food/dining) associated with aconcert that will be occurring at a specified location and future time.The relevant inputs received at 560 are actual ticket sales data (A) forthe concert, social media data (B) associated with the concert (e.g.,Twitter® tweets, Instagram posts and/or Facebook® posts about theconcert, etc.), actual prior attendance data (C) for a similar concert(e.g., a prior occurrence of a named music festival), weather forecastdata (D) for the date on which the concert will take place, and venuesize data (E) for the venue where the concert will take place.Intermediate predictions of concert attendance (562A) and one or moretelecommunications metrics (562B) associated with the concert aregenerated, based on one or more of the inputs received at 560, duringthe first prediction phase 562. Based on the prediction of concertattendance generated at 562A and/or the one or more telecommunicationsmetrics generated at 562B, one or more of the following predictions ofdemand are generated during the second prediction phase 564:telecommunications usage data (564A), local traffic (564B), sales for alocal restaurant (564C), emergency services (564D), and merchandisesales for a vendor (564E). Each of the predictions of demand generatedduring the second prediction phase 564 is associated with the concert,and thus with the geographic location and period of time associated withthat concert. The predictions of demand generated during the secondprediction phase 564 can be used to inform resource allocation decisionsfor the concert (e.g., a telecommunications provider may decide to makeadditional bandwidth available for that geographic region during theconcert period, local traffic enforcement may staff intersections withinthe geographic region associated with the concert during the concertperiod, and/or the local restaurant may increase inventory and/orstaffing during the concert period).

Time Series Sampling

In some embodiments, the sampling frequency of a signal has asignificant impact on its identification. Datasets often include timeseries data that has been sampled at non-uniform sampling intervals,however many prediction methods are designed to process data that hasbeen sampled at uniform partitions of time. As such, a resampling oftime series data of dataset is often needed. The resampling rate can bedetermined by the frequency at which the predictions will be sampled bythe end user of the forecast. In large-scale population volume andactivity prediction, it is often the case that the signals determiningpopulation movement are sampled at much higher frequencies than thefrequency at which the end user will query the forecasts. For example,telecommunications datasets provide information about the location andlatency of cellular devices with an average sampling rate of seconds, orminutes, much higher than the 1-hour forecast resolution that isrequired for practical use.

Down-sampling aggregates the higher sampling frequency of the series toa lower frequency by aggregating a summary statistic. The mean of thehigher frequencies can be used, however in other implementations, themaximum of the higher frequencies or the sum of the higher frequenciescan be used.

Structure Exploration

It can be desirable, when resampling the time series, to use a samplingfrequency that is high enough to capture all or most underlying signals,while assuming that a signal cannot necessarily be cleanlyseparated/filtered from the noise that accompanies it (e.g., including afixed, finite sum of frequencies). In some implementations, it isassumed that the underlying statistical model parameters of the timeseries remain stationary through time.

Auto-Regressive Moving Average (ARMA) Model Definition

One approach to time series modeling is referred to as an AutoregressiveMoving Average (ARMA) process. In ARMA modeling, a next time step of aprocess is generated as a linear combination of previous values andprevious errors. An ARMA(p,q) process of autoregressive (AR) order p andmoving average (MA) order q for a stochastic process {X_(t)} is givenby:X _(t) =+a ₁ X _(t−1) + . . . +a _(p) X _(t−p)+ε_(t)+θ₁ε_(t−1)+ . . .+θ_(q)ε_(t31 q)where ε_(i)˜N(0, σ²) are independent error/noise terms, and the “a”values are model parameters. Here we assume that X_(i) is centered atzero by adjusting the intercept coefficient a appropriately. A moreconvenient way of representing this model using the backshift operatorB^(p) (X_(t))=X_(t−p) is given by the following expression, in which Bis a backshift operator or lag operator that operates on an element ofthe time-series to produce the previous element:(1−a ₁ B− . . . −a _(p) B ^(p))X _(t) =a+(1+θ₁ B+ . . . +θ _(q) B^(q))ε_(t)

Notice that the error terms ε_(i) are unobserved. In terms of thedynamics of the model, these error terms correspond to ‘corrections’ inthe system based on previous errors, and are responsive to shocks in thesystem.

Stationarity and Differencing

It is not necessarily the case that the underlying statistical modelparameters of the time series remain stationary through time. A serieswith a steady upward trend, for example, exhibits a time-dependent mean,which can be accommodated using a differencing process (describedbelow). Alternatively or in addition, a series may have a time-dependentvariance.

The first difference of a time series X_(t) is given by X_(t)−X_(t−1).If X_(t) exhibits a non-exponential upward or downward trend, then theseries given by its first difference will likely not exhibit the sametrend, as the differences in values are steady. If the first differenceis not sufficient to introduce stationarity (“stationarity” referring toa condition in which the differences in values are steady, or that themean and/or variance remain steady/substantially constant over time), asecond difference may be implemented, however, over-differencing comesat the price of lower forecasting accuracy. If the differenced timeseries is forecast, the forecasts may be added to the differenced seriesX_(t)−X_(t−1) to recover forecasts for X_(t).

Autocorrelation Plots (ACF) and Partial Autocorrelation Plots (PACF)

According to the Box-Jenkins approach to automatic ARMA model selection,Autocorrelation and Partial Autocorrelation plots can be implemented toidentify the appropriate AR and MA orders. A detailed theoreticalexposition on the plots and their estimations is given in Chapter 3 ofShumway & Stoffer's Time Series and Its Applications. Example rules forthe selection of AR and MA structures are summarized as follows:

If the PACF of the differenced series displays a sharp cutoff and/or thelag-1 autocorrelation is positive—i.e., if the series appears slightly“underdifferenced”—then consider adding an AR term to the model. The lagat which the PACF cuts off is the indicated number of AR terms.If the ACF of the differenced series displays a sharp cutoff and/or thelag-1 autocorrelation is negative—i.e., if the series appears slightly“overdifferenced”—then consider adding an MA term to the model. The lagat which the ACF cuts off is the indicated number of MA terms.Seasonality

Seasonality may be incorporated to the model by explicit introduction ofadditional seasonal terms in the linear combination. As an example, anARMA(1,1)(1,1) model for a dataset with 24 observations per season, withseasonal autoregressive order P and seasonal moving average order M isgiven by:(1−a ₁ B− . . . −a _(p) B ^(p))(1−A ₁ B ²⁴ − . . . −A _(p) B ^(24 P))X_(t) =a+(1+θ₁ B+ . . . +θ _(q) B ^(q))(1+Θ₁ B ²⁴+ . . . +Θ_(Q) B^(24 Q))ε_(t)Fourier Terms and Power Spectral Density

An alternative way of introducing seasonality to the model is throughthe insertion of a superimposed sinusoid with period corresponding tothe seasonality of the model. This is especially useful in capturingmultiple seasonality present in the series. We can combine the Fourierterms with an Autoregressive Integrated Moving Average (ARIMA) processvia:

$X_{t} = {a + {\sum\limits_{i = 1}^{M}{\sum\limits_{k = 1}^{K_{i}}\left\lbrack {{a_{i}{\sin\left( \frac{2\pi kt}{p_{i}} \right)}} + {b_{i}{\cos\left( \frac{2\pi kt}{p_{i}} \right)}}} \right\rbrack}} + {A\; R\; I\; M\; A}}$where there are M different seasonalities with periods p_(i) and K_(i)Fourier terms each. If seasonality is not evident either in the timeplots or the correlation plots, then a periodogram of the power spectraldensity of the signal can be used to filter out the frequencies ofhighest ‘power’, which in turn can be used to fit the appropriateFourier terms or seasonal autoregressive terms to the series to captureits seasonality.Additional Features

Once a standard ARIMA structure has been determined, additional featurescan be investigated. In some instances, ARIMA does not capture all ofthe dynamics of the underlying process. By using error analysis,discussed below, AutoCorrelation Function (ACF)/Partial AutoCorrelationFunction (PACF) plots and histograms can help identify underlyingstructures that ARIMA failed to capture. Root Mean Square Error (RMSE)or other measures of accuracy can be improved by the introduction ofnon-standard features specific to the dataset from which the series wasproduced, in some embodiments.

For a time series having an hourly or daily resolution, ‘time of day’and ‘day of week’ information can prove valuable, and may be introducedto the model via one-hot encoded dummy variables. Care should be takenwhen one-hot encoding such variables not to introduce linear dependencyamong the columns. When Ordinary Least Squares (OLS) is implemented forthe regression, an intercept is excluded at model specification. Anexample of such introduction of dummy variables for ‘day of week’information is given by the following ARIMA structure plus the one-hotencoded variables representing the days of the week.

$X_{t} = {{\sum\limits_{i = 1}^{p}{a_{i}X_{t - i}}} + {\sum\limits_{i = 1}^{q}{\theta_{i}\epsilon_{t - i}}} + {\sum\limits_{i = 1}^{7}{d_{i}D_{i}}} + \epsilon_{t}}$where D_(i) represents the i-th day of the week and takes the value 1 ifX_(t) is in day D_(i), and 0 otherwise. Notice that the interceptcoefficient is not included in the model.

Additional features that can be taken into account are the maxima of aseries and the minima of a series, as well as the sum totals, within aspecified time window. When such additional features are incorporated tothe model, a similar approach can be taken in the regression equation byadding additional variables whose values are piecewise determined bytheir intended use. For example, to incorporate the maximum of the last30 observations in the model, we define:

$X_{t} = {{\sum\limits_{i = 1}^{p}{a_{i}X_{t - i}}} + {\sum\limits_{i = 1}^{q}{\theta_{i}\epsilon_{t - i}}} + {b_{i}M_{i}}}$where

$M_{i} = {\max\limits_{t \geq {T - 30}}{X_{t}.}}$Care should be taken when considering additional features not to addvariables that will be linearly dependent to the other inputs of themodel. For example, adding the mean of the series over a specified timewindow as a new feature may introduce dependency since the mean is alinear combination of previous X_(t) values that already appear in theautoregressive part of the model.External Series

In some embodiments, the ARIMA structure and the additional featuresdiscussed above can be extracted directly from the historical values ofthe time series being forecast. They are incorporated into the model viathe construction of a new time series that stands as a new variable inthe linear combination describing the next timestep. If externalinformation is provided in the form of a separate time series Z_(t) withthe same evenly spaced time steps as X_(t), it can similarly beincorporated as a predictor.

Depending on the availability of the data from Z_(t) there are threeways such a series can be used to forecast X_(t). If the series of Z_(t)is generated at the same time as the series of X_(t), then the next timestep of Z_(t) is unknown, and we have the option to either use Z_(t) asa lagged predictor via:X _(t+1) =a ₁ Z _(t)+ARMA+additional featuresor we can forecast Z_(t) and use the forecasted value via:X _(t+1) =a ₁ F(Z _(t))+ARMA+additional featureswhere F(Z_(t)) is the forecasted value for Z_(t+1). Alternatively, ifZ_(t) is available beyond the time step we are forecasting X_(t) for,then we can directly substitute the next time step in the model:X _(t+1) =a ₁ Z _(t+1)+ARMA+additional features

At this stage, a regression or a maximum likelihood estimate can beimplemented to fit the model to the data.

External Series—Use Case

In some embodiments, a knowledge base on population movement includesdata from a wide range of sources, including but not limited to:

-   -   a) Event data, e.g., organized and pre-planned events such as        sporting events, concerts, conferences, exhibitions etc.    -   b) Historical data on locations of persistent population        concentration and interest, e.g., bars, cafés, clubs, hospitals,        train stations, etc.    -   c) Predictive data relevant to population movement, e.g.,        weather forecasts and historical traffic reports.    -   d) Live data, e.g., social media data and data associated with        transportation service emergencies.    -   e) Static data, e.g., a number of businesses, venue capacity and        demographics

Using this information, absolute and relative measures of populationvolume can be determined. For example, event data sources (a) can becombined with venue capacity information from static data (e) togenerate a population volume series V_(t) for any locality at time t.

Further, data from locations of interest (b) provide normalizedhistorical averages, maxima and minima of attendance at bars, cafés,restaurants and other points of interest throughout a locality, whichcombined with venue capacity information can also provide a relativizedpopulation volume series V _(t). A normalization, re-factoring andappropriate re-sampling step allows absolute population volume estimatesV_(t) to be combined with relative population volume estimates V _(t) togenerate comprehensive population volume estimates Z_(t) over time.

As described in the previous section, this time series Z_(t) can be usedas a significant predictor in any process that is correlated withpopulation volume (e.g., crime frequency and volume, mobiletelecommunication latency, etc.). To generate models for a process thatis causal or correlated with population volume, analysis of the internalstructure of the process as set forth herein can be combined withappropriately normalized and localized population volume time series ina scheme that can be roughly described via:X _(t)=Population Series+ARIMA+Additional internal features+FouriertermsHypothesis (Setting Model Hyperparameters)

In some embodiments, the steps undertaken in structure exploration asdescribed herein are used to create a hypothesis about the underlyingdynamics guiding/driving the time series. If there is an autoregressiveor moving average structure, the appropriate ARMA orders can bespecified and included in the model. If there is a critical saturationlevel that the series may reach, then the sum totals over a fixed timewindow may be included. If a clear seasonal component is present, aFourier term can be introduced to fit it appropriately.

It should be noted that the feature selection/extraction process and themodel hypothesis can be implemented in an iterated fashion. For example,domain knowledge can be used to form a model hypothesis that informs afeature selection process. Alternatively, a statistically-driven featureselection and error analysis can be used to generate a hypothesis.

A model can be defined as a linear combination of the underlyingparameters, represented in vector form by:y=Xbwhere X is a matrix of observations (rows) for each of the variables inour model (columns), with target variables (observations) y, andcoefficients b to be estimated from the observations.ForecastingEstimation

In some embodiments, a baseline approach for coefficient estimation isan OLS estimate for the minimization problem, as follows:b=argmin∥y−X b∥

A linear regression in Python's scikit-learn can be implemented for thetask of minimizing the errors ∥y−X b∥. This yields a baseline root meansquared error (RMSE), mean average percentage error (MAPE), mean averagesquare error (MASE), R2 and Akaike Information Criterion (AIC_C) scores.

Two techniques for linear regression are implemented Ordinary LeastSquares (OLS) and Decision Trees, which may be sufficient for mosttasks. In some implementations, however, if it is desirable to addMoving Average terms, unobserved error terms may be added to the linearcombination of the model. Since these terms are unobserved, a MaximumLikelihood Estimate may be implemented. Statsmodels' ARMA functionprovides ARMA models with maximum likelihood estimation (MLE)implementations for moving average terms.

Regression estimates that improve on OLS include Ridge and LassoRegression. These may similarly be implemented via scikit-learn'sRegression library. An alternative estimator for the same underlyinglinear combination model is a decision tree. As shown via theimplemented examples below, the two estimates yield very similarresults, with OLS having a slight advantage. Decision trees, however,offer greater flexibility in the form of random forests and gradientboosting. Decision Tree Ensembles, such as the xgboost algorithm havegained popularity through their successful use in forecastingcompetitions, along with built-in implementations in the Pythonecosystem.

Training/Test Splits

A traditional train and test set split was used for the training andevaluation of the model. The time-dependence of the data did not permitthe use of a fully-fledged cross-validation schema, however, a slightimprovement on standard train/test splits is given by the followingfolding structure, in which a time series is divided into six parts [1],. . . , [6], as follows:

Fold  1:  train  [1] − test  [2] Fold  2:  train  [1], [2] − test  [3] ⋮Fold  5:  train  [1], …  , [4] − test  [5]Single-Step

Models can be compared, for example, on the basis of their single-stepforecasting accuracy. In other words, for every point in a test set, asingle-point forecast is produced. The residuals resulting from thissingle-step prediction can be used to produce RMSE, MAPE and AIC_Cscores, which form the foundation for cross-model comparison.

Multi-Step

In some applications, multi-step forecasts are desired. A regressionmodel may be equipped to predict only a single step in the future basedon historical values, however if multiple future values are desired, aniterated approach can be taken, in which each prediction is taken as aground truth and used along with previous historical values to generatea new prediction for the following time step. The model can optionallybe re-fit based on this new prediction value.

Error Analysis

Residuals

In some embodiments, the residuals of a model that sufficiently capturesthe dynamics of a process are preferably normally distributed andcentered at zero. Further, the autocorrelation and partialautocorrelation plots of the residuals preferably show no significant ARor MA structure. That is, it is preferred that there be limited or nolinear dependence between consecutive residuals, and no seasonal patternto their occurrence.

If the data is normally distributed but not centered at zero, there maybe bias or other unexplained error structure(s) in the model that skewsthe outputs consistently above or below center. A solution to this skewcan include fitting an ARMA model to the residual series based on any ARor MA signatures we see in the residuals' ACF and PACF plots

Error Metrics

Different types of error measurement can be used, depending for exampleon the desired accuracy of the predictions. For example, the followingmetrics can be employed on the test set:

-   -   a) Root Mean Squared Error (RMSE)—the standard Euclidean        distance of a vector from zero. This distance is biased towards        outliers resulting from squaring. The square root is applied to        make the units of the errors same as the units of the        observations. This may be preferred when measuring the average        number of units that differentiate the forecasts from associated        real observations. The root may be omitted when the objective is        only to minimize this error.    -   b) Mean Absolute Percentage Error (MAPE)—an average of the        percentage by which each prediction misses its corresponding        observation. This is a direct way of retrieving a percentage of        the forecasting accuracy.    -   c) Mean Absolute Scaled Error (MASE)—the average of the ratios        of absolute forecast errors against the mean absolute error of a        naïve forecast. A naïve forecast may be defined as a forecast        that uses the previous time step as a forecast for the next time        step. If this metric is less than 1, then the model is        out-performing the naïve forecast.

All combinations of the foregoing concepts and additional conceptsdiscussed herewithin (provided such concepts are not mutuallyinconsistent) are contemplated as being part of the subject matterdisclosed herein. The terminology explicitly employed herein that alsomay appear in any disclosure incorporated by reference should beaccorded a meaning most consistent with the particular conceptsdisclosed herein.

The skilled artisan will understand that the drawings primarily are forillustrative purposes, and are not intended to limit the scope of thesubject matter described herein. The drawings are not necessarily toscale; in some instances, various aspects of the subject matterdisclosed herein may be shown exaggerated or enlarged in the drawings tofacilitate an understanding of different features. In the drawings, likereference characters generally refer to like features (e.g.,functionally similar and/or structurally similar elements).

The term “automatically” is used herein to modify actions that occurwithout direct input or prompting by an external source such as a user.Automatically occurring actions can occur periodically, sporadically, inresponse to a detected event (e.g., a user logging in), or according toa predetermined schedule.

The term “determining” encompasses a wide variety of actions and,therefore, “determining” can include calculating, computing, processing,deriving, investigating, looking up (e.g., looking up in a table, adatabase or another data structure), ascertaining and the like. Also,“determining” can include receiving (e.g., receiving information),accessing (e.g., accessing data in a memory) and the like. Also,“determining” can include resolving, selecting, choosing, establishingand the like.

The phrase “based on” does not mean “based only on,” unless expresslyspecified otherwise. In other words, the phrase “based on” describesboth “based only on” and “based at least on.”

The term “processor” should be interpreted broadly to encompass ageneral purpose processor, a central processing unit (CPU), amicroprocessor, a digital signal processor (DSP), a controller, amicrocontroller, a state machine and so forth. Under some circumstances,a “processor” may refer to an application specific integrated circuit(ASIC), a programmable logic device (PLD), a field programmable gatearray (FPGA), etc. The term “processor” may refer to a combination ofprocessing devices, e.g., a combination of a DSP and a microprocessor, aplurality of microprocessors, one or more microprocessors in conjunctionwith a DSP core or any other such configuration.

The term “memory” should be interpreted broadly to encompass anyelectronic component capable of storing electronic information. The termmemory may refer to various types of processor-readable media such asrandom access memory (RAM), read-only memory (ROM), non-volatile randomaccess memory (NVRAM), programmable read-only memory (PROM), erasableprogrammable read only memory (EPROM), electrically erasable PROM(EEPROM), flash memory, magnetic or optical data storage, registers,etc. Memory is said to be in electronic communication with a processorif the processor can read information from and/or write information tothe memory. Memory that is integral to a processor is in electroniccommunication with the processor.

The terms “instructions” and “code” should be interpreted broadly toinclude any type of computer-readable statement(s). For example, theterms “instructions” and “code” may refer to one or more programs,routines, sub-routines, functions, procedures, etc. “Instructions” and“code” may comprise a single computer-readable statement or manycomputer-readable statements.

Some embodiments described herein relate to a computer storage productwith a non-transitory computer-readable medium (also can be referred toas a non-transitory processor-readable medium) having instructions orcomputer code thereon for performing various computer-implementedoperations. The computer-readable medium (or processor-readable medium)is non-transitory in the sense that it does not include transitorypropagating signals per se (e.g., a propagating electromagnetic wavecarrying information on a transmission medium such as space or a cable).The media and computer code (also can be referred to as code) may bethose designed and constructed for the specific purpose or purposes.Examples of non-transitory computer-readable media include, but are notlimited to, magnetic storage media such as hard disks, floppy disks, andmagnetic tape; optical storage media such as Compact Disc/Digital VideoDiscs (CD/DVDs), Compact Disc-Read Only Memories (CD-ROMs), andholographic devices; magneto-optical storage media such as opticaldisks; carrier wave signal processing modules; and hardware devices thatare specially configured to store and execute program code, such asApplication-Specific Integrated Circuits (ASICs), Programmable LogicDevices (PLDs), Read-Only Memory (ROM) and Random-Access Memory (RAM)devices. Other embodiments described herein relate to a computer programproduct, which can include, for example, the instructions and/orcomputer code discussed herein.

Some embodiments and/or methods described herein can be performed bysoftware (executed on hardware), hardware, or a combination thereof.Hardware modules may include, for example, a general-purpose processor,a field programmable gate array (FPGA), and/or an application specificintegrated circuit (ASIC). Software modules (executed on hardware) canbe expressed in a variety of software languages (e.g., computer code),including Python, C, C++, Java™, Ruby, Visual Basic™, and/or otherobject-oriented, procedural, or other programming language anddevelopment tools. Examples of computer code include, but are notlimited to, micro-code or micro-instructions, machine instructions, suchas produced by a compiler, code used to produce a web service, and filescontaining higher-level instructions that are executed by a computerusing an interpreter. For example, embodiments may be implemented usingimperative programming languages (e.g., C, Fortran, etc.), functionalprogramming languages (Haskell, Erlang, etc.), logical programminglanguages (e.g., Prolog), object-oriented programming languages (e.g.,Java, C++, etc.) or other suitable programming languages and/ordevelopment tools. Additional examples of computer code include, but arenot limited to, control signals, encrypted code, and compressed code.

Various concepts may be embodied as one or more methods, of which atleast one example has been provided. The acts performed as part of themethod may be ordered in any suitable way. Accordingly, embodiments maybe constructed in which acts are performed in an order different thanillustrated, which may include performing some acts simultaneously, eventhough shown as sequential acts in illustrative embodiments. Putdifferently, it is to be understood that such features may notnecessarily be limited to a particular order of execution, but rather,any number of threads, processes, services, servers, and/or the likethat may execute serially, asynchronously, concurrently, in parallel,simultaneously, synchronously, and/or the like in a manner consistentwith the disclosure. As such, some of these features may be mutuallycontradictory, in that they cannot be simultaneously present in a singleembodiment. Similarly, some features are applicable to one aspect of theinnovations, and inapplicable to others.

The indefinite articles “a” and “an,” as used herein in thespecification and in the embodiments, unless clearly indicated to thecontrary, should be understood to mean “at least one.”

The phrase “and/or,” as used herein in the specification and in theembodiments, should be understood to mean “either or both” of theelements so conjoined, i.e., elements that are conjunctively present insome cases and disjunctively present in other cases. Multiple elementslisted with “and/or” should be construed in the same fashion, i.e., “oneor more” of the elements so conjoined. Other elements may optionally bepresent other than the elements specifically identified by the “and/or”clause, whether related or unrelated to those elements specificallyidentified. Thus, as a non-limiting example, a reference to “A and/orB”, when used in conjunction with open-ended language such as“comprising” can refer, in one embodiment, to A only (optionallyincluding elements other than B); in another embodiment, to B only(optionally including elements other than A); in yet another embodiment,to both A and B (optionally including other elements); etc.

As used herein in the specification and in the embodiments, “or” shouldbe understood to have the same meaning as “and/or” as defined above. Forexample, when separating items in a list, “or” or “and/or” shall beinterpreted as being inclusive, i.e., the inclusion of at least one, butalso including more than one, of a number or list of elements, and,optionally, additional unlisted items. Only terms clearly indicated tothe contrary, such as “only one of” or “exactly one of,” or, when usedin the embodiments, “consisting of,” will refer to the inclusion ofexactly one element of a number or list of elements. In general, theterm “or” as used herein shall only be interpreted as indicatingexclusive alternatives (i.e. “one or the other but not both”) whenpreceded by terms of exclusivity, such as “either,” “one of,” “only oneof,” or “exactly one of.” “Consisting essentially of,” when used in theembodiments, shall have its ordinary meaning as used in the field ofpatent law.

In the embodiments, as well as in the specification above, alltransitional phrases such as “comprising,” “including,” “carrying,”“having,” “containing,” “involving,” “holding,” “composed of,” and thelike are to be understood to be open-ended, i.e., to mean including butnot limited to. Only the transitional phrases “consisting of” and“consisting essentially of” shall be closed or semi-closed transitionalphrases, respectively, as set forth in the United States Patent OfficeManual of Patent Examining Procedures, Section 2111.03.

While specific embodiments of the present disclosure have been outlinedabove, many alternatives, modifications, and variations will be apparentto those skilled in the art. Accordingly, the embodiments set forthherein are intended to be illustrative, not limiting. Various changesmay be made without departing from the spirit and scope of thedisclosure.

The invention claimed is:
 1. A method, comprising: receiving, via aprocessor, site data for a plurality of geographic areas; receiving, viathe processor, event data for each geographic area from the plurality ofgeographic areas, the event data including movement data associated withat least one of an individual or a group of individuals; generating, viathe processor, a predicted resource demand for each geographic area fromthe plurality of geographic areas and for each predetermined future timeinterval from a plurality of predetermined future time intervals, basedon the site data and the event data; and in response to receiving anindication of a selected predetermined geographic area from theplurality of predetermined geographic areas and an indication of aselected predetermined future time interval from the plurality ofpredetermined future time intervals via at least one of a GUI, an API,or a CSV file, sending a signal to cause at least one of: display of thepredicted resource demand via the GUI, delivery of the predictedresource demand via an API, or delivery of the predicted resource demandvia an associated CSV file.
 2. The method of claim 1, furthercomprising: generating, via the processor, a recommended action for aride-hailing system, based on the predicted resource demand, wherein therecommended action for the ride-hailing system is associated with oneof: a vehicle availability, a driver incentive, or a passenger pricingadjustment.
 3. The method of claim 1, wherein the predicted resourcedemand is a quantity of line items.
 4. The method of claim 1, whereinthe predicted resource demand is a demand for staffing.
 5. The method ofclaim 1, wherein the predicted resource demand is a demand for energy.6. The method of claim 1, wherein the predicted resource demand is ademand for transportation.
 7. The method of claim 1, wherein thepredicted resource demand is a demand for self-driving cars.
 8. Themethod of claim 1, wherein the generating the predicted resource demandis performed using time series modeling.
 9. The method of claim 1,wherein the generating the predicted resource demand is performed usinga domain-specific machine learning algorithm.
 10. The method of claim 9,wherein the domain-specific machine learning algorithm is a machinelearning algorithm configured to generate predictions specific to oneof: ride hailing, self-driving cars, food supply, or staffing.
 11. Themethod of claim 1, wherein at least one of the receiving the site dataor the receiving the event data is via at least one of: the GUI, theAPI, or an associated CSV file.
 12. A system, comprising: a processor;and a memory storing processor-readable instructions to cause theprocessor to: receive site data for a plurality of geographic areas;receive event data for each geographic area from the plurality ofgeographic areas, the event data including movement data associated withat least one of an individual or a group of individuals; generate apredicted resource demand for each geographic area from the plurality ofgeographic areas and for each predetermined future time interval from aplurality of predetermined future time intervals, based on the site dataand the event data; and in response to receiving an indication of aselected predetermined geographic area from the plurality ofpredetermined geographic areas and an indication of a selectedpredetermined future time interval from the plurality of predeterminedfuture time intervals via at least one of a GUI, an API, or a CSV file,send a signal to cause at least one of: display of the predictedresource demand via the GUI, delivery of the predicted resource demandvia an API, or delivery of the predicted resource demand via anassociated CSV file.
 13. The system of claim 12, wherein the memoryfurther stored processor-readable instructions to cause the processorto: generate a recommended action for a ride-hailing system, based onthe predicted resource demand, wherein the recommended action for theride-hailing system is associated with one of: a vehicle availability, adriver incentive, or a passenger pricing adjustment.
 14. The system ofclaim 12, wherein the predicted resource demand is a quantity of lineitems.
 15. The system of claim 12, wherein the predicted resource demandis a demand for staffing.
 16. The system of claim 12, wherein thepredicted resource demand is a demand for energy.
 17. The system ofclaim 12, wherein the predicted resource demand is a demand fortransportation.
 18. The system of claim 12, wherein the predictedresource demand is a demand for self-driving cars.
 19. The system ofclaim 12, wherein the instructions to cause the processor to generatethe predicted resource demand include instructions to perform timeseries modeling.
 20. The system of claim 12, wherein the instructions tocause the processor to generate the predicted resource demand includeinstructions to generate the predicted resource demand using adomain-specific machine learning algorithm.
 21. The system of claim 20,wherein the domain-specific machine learning algorithm is a machinelearning algorithm configured to generate predictions specific to oneof: ride hailing, self-driving cars, food supply, or staffing.
 22. Thesystem of claim 12, wherein the instructions to cause the processor toreceive the site data for the plurality of geographic areas includeinstructions to cause the processor to receive the site data via atleast one of: the GUI, the API, or an associated CSV file.